Electronic apparatus and control method thereof

ABSTRACT

An electronic apparatus is provided. The electronic apparatus according to the disclosure may include a memory configured to store instructions, and a processor configured to execute the instructions to control a display to display an image captured via a camera, and a first guideline, recognize an object in the image based on the first guideline, render a virtual object to replace the object, generate an augmented reality image including the virtual object that is located in an area where the object is located in the image; and control the display to display the augmented reality image.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is based on and claims priority under 35 U.S.C. § 119to U.S. Provisional Patent Application No. 62/882,106, filed on Aug. 2,2019, in the U.S. Patent & Trademark Office, and Korean PatentApplication No. 10-2019-0172247, filed on Dec. 20, 2019, in the KoreanIntellectual Property Office, the disclosures of which are incorporatedby reference herein in their entireties.

BACKGROUND 1. Field

The disclosure relates to an electronic apparatus and a control methodthereof, and more particularly, to an electronic apparatus displayingaugmented reality images, and a control method thereof.

2. Description of Related Art

Augmented reality is a type of virtual reality which combines the realworld seen by a user and a virtual world, and shows the combination as acombined image. Augmented reality, which is a concept of complementingthe real word with a virtual world, uses a virtual environment generatedby computer graphics, but is based on the real environment. Here,computer graphics perform the role of additionally providing images orinformation to augment the real environment. That is, computer graphicsoverlap a three-dimensional virtual image with an actual image that auser is viewing, and thereby make reduce a distinction between the realenvironment and the virtual screen.

In this regard, a technology of rendering a virtual object in an emptyspace of an actual image by using an augmented reality technology isbeing generalized. In a case in which an object exists in an actualimage, a complex operation and an apparatus with a high technicalspecification are required for rendering a virtual object in the placewherein the object is located. Also, a problem exists, which is that thesize of a space wherein an object is located cannot be preciselyidentified according to the photographing angle, the photographinglocation, etc. even though a complex operation was performed.

SUMMARY

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments.

According to an aspect of the disclosure, an electronic apparatus mayinclude a display, a memory configured to store instructions, and aprocessor configured to execute the instructions to control a display todisplay an image captured via a camera, and a first guideline, recognizean object in the image based on the first guideline, render a virtualobject to replace the object, generate an augmented reality imageincluding the virtual object that is located in an area where the objectis located in the image; and control the display to display theaugmented reality image.

Based on an event, the processor may render the virtual object toindicate a movement of the virtual object.

The processor may identify a length of a blank space existing in anadjacent area of the object, identify whether the virtual object iscapable of being located in the area where the object is located basedon length information of the object and the length of the blank space,and based on the virtual object being capable of being located in thearea where the object is located, render the virtual object.

The processor may identify whether the virtual object is capable ofcovering an object area of the object in the image, and generate theaugmented reality image by replacing a partial object area of the objectthat is not covered by the virtual object with a background areaadjacent to the partial object area.

The processor may control the display to display a plurality of virtualobjects that are capable of replacing the object, and based on a userinput selecting one of the plurality of virtual objects being received,render the virtual object corresponding to the user input.

Based on a line being input in association with a surface of the objectin the image, the processor may determine a location of the virtualobject in the augmented reality image based on the input line.

The processor may identify a location of the virtual object in theaugmented reality image based on location information and angleinformation of the camera, and generate the augmented reality image byarranging the virtual object in the location.

The processor may generate a second guideline for guiding a capturingangle of the camera, and change a length of the second guidelineaccording to the capturing angle of the camera.

Based on receiving a user input that moves the virtual object includedin the augmented reality image, the processor may change a location ofthe virtual object based on the user input and render the virtual objectaccording to the changed location.

Based on location information or angle information of the camera beingchanged while the augmented reality image is displayed, the processormay change the virtual object based on the location information or angleinformation of the camera.

According to an aspect of the disclosure, a control method of anelectronic apparatus may include capturing an image; displaying a firstguideline and the image; recognizing an object in the image based on thefirst guideline; rendering a virtual object to replace the object;generating an augmented reality image including the virtual object thatis located in an area where the object is located in the image; anddisplaying the augmented reality image.

Based on an event, the control method may further include rendering thevirtual object to indicate a movement of the virtual object.

The rendering a virtual object may include identify a length of a blankspace existing in an adjacent area of the object, identifying whetherthe virtual object is capable of being located in the area where theobject is located based on length information of the object and thelength of the blank space, and based on the virtual object being capableof being located in the area where the object is located, rendering thevirtual object.

The generating an augmented reality image may include identifyingwhether the virtual object is capable of covering an object area of theobject in the image, and generating the augmented reality image byreplacing a partial object area of the object that is not covered by thevirtual object with a background area adjacent to the partial objectarea.

The control method may further include displaying a plurality of virtualobjects that are capable of replacing the object, and based on a userinput selecting one of the plurality of virtual objects being received,rendering the virtual object corresponding to the user input.

Based on a line being input in association with a surface of the objectin the image, the control method may further include determining alocation of the virtual object in the augmented reality image based onthe input line.

The control method may further include identifying a location of thevirtual object in the augmented reality image based on locationinformation and angle information of a camera, and generating theaugmented reality image by arranging the virtual object in the location.

The control method may further include generating a second guideline forguiding a capturing angle of a camera, and changing a length of thesecond guideline according to the capturing angle of the camera.

The control method may further include receiving a user input that movesthe virtual object included in the augmented reality image, and changinga location of the virtual object based on the user input and renderingthe virtual object according to the changed location.

According to an aspect of the disclosure, a non-transitorycomputer-readable medium may store instructions that, when executed by aprocessor of an apparatus, cause the processor to control a display ofthe apparatus to display an image captured via a camera of theapparatus, control the display to display a guideline in the image,recognize the object in the image based on the first guideline beingaligned with the object, generate an augmented reality image including avirtual object having a location that corresponds to a location of theobject in the image, and control the display to display the augmentedreality image.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the present disclosure will be more apparent from thefollowing description taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 is a diagram for schematically illustrating an electronicapparatus according to an embodiment;

FIG. 2 is a flow chart for illustrating a control method of anelectronic apparatus according to an embodiment;

FIG. 3 is a block diagram for illustrating a configuration of anelectronic apparatus according to an embodiment;

FIG. 4A is a diagram for illustrating an electronic apparatus indicatinga guideline according to an embodiment;

FIG. 4B is a diagram for illustrating an electronic apparatus indicatinga guideline according to an embodiment;

FIG. 4C is a diagram for illustrating an electronic apparatus indicatinga guideline according to an embodiment;

FIG. 5A is a diagram for illustrating an electronic apparatus indicatinga guideline according to an embodiment;

FIG. 5B is a diagram for illustrating an electronic apparatus indicatinga guideline according to an embodiment;

FIG. 6A is a diagram for illustrating an electronic apparatus indicatinga plurality of virtual objects according to an embodiment;

FIG. 6B is a diagram for illustrating an electronic apparatus indicatinga plurality of virtual objects according to an embodiment;

FIG. 7A is a diagram for illustrating an electronic apparatusdetermining a blank space and rendering a virtual object according to anembodiment;

FIG. 7B is a diagram for illustrating an electronic apparatusdetermining a blank space and rendering a virtual object according to anembodiment;

FIG. 7C is a diagram for illustrating an electronic apparatusdetermining a blank space and rendering a virtual object according to anembodiment;

FIG. 7D is a diagram for illustrating an electronic apparatusdetermining a blank space and rendering a virtual object according to anembodiment;

FIG. 8A is a diagram for illustrating a method of determining thelocation of a virtual object according to an embodiment;

FIG. 8B is a diagram for illustrating a method of determining thelocation of a virtual object according to an embodiment;

FIG. 9A is a diagram for illustrating an electronic apparatus generatingan augmented reality image by using a surrounding background areaaccording to an embodiment;

FIG. 9B is a diagram for illustrating an electronic apparatus generatingan augmented reality image by using a surrounding background areaaccording to an embodiment;

FIG. 10A is a diagram for illustrating an electronic apparatus changinga virtual object according to the location of a camera and rendering thevirtual object according to an embodiment;

FIG. 10B is a diagram for illustrating an electronic apparatus changinga virtual object according to the location of a camera and rendering thevirtual object according to an embodiment;

FIG. 11A is a diagram for illustrating an electronic apparatus renderinga virtual object indicating a movement during a driving operation of avirtual object according to an embodiment;

FIG. 11B is a diagram for illustrating an electronic apparatus renderinga virtual object indicating a movement during a driving operation of avirtual object according to an embodiment;

FIG. 12A is a diagram for illustrating an electronic apparatus changingthe location of a virtual object according to a user input moving avirtual object included in an augmented reality image and rendering thevirtual object according to an embodiment;

FIG. 12B is a diagram for illustrating an electronic apparatus changingthe location of a virtual object according to a user input moving avirtual object included in an augmented reality image and rendering thevirtual object according to an embodiment; and

FIG. 13 is a block diagram for illustrating a detailed configuration ofan electronic apparatus according to an embodiment.

DETAILED DESCRIPTION

The example embodiments of the disclosure provide an electronicapparatus which improves convenience and accuracy of manipulation byproviding a guide to a user, reduces operation complexity, and therebyprovides an augmented reality image with a low specification, and acontrol method thereof.

Hereinafter, various embodiments of the disclosure will be describedwith reference to the accompanying drawings. However, it should be notedthat the various embodiments are not for limiting the technologydescribed in the disclosure to a specific embodiment, but they should beinterpreted to include various modifications, equivalents, and/oralternatives of the embodiments of the disclosure. Also, with respect tothe detailed description of the drawings, similar components may bedesignated by similar reference numerals.

In the disclosure, expressions such as “have,” “may have,” “include,”and “may include” should be construed as denoting that there are suchcharacteristics (e.g., elements such as numerical values, functions,operations, and components), and the expressions are not intended toexclude the existence of additional characteristics.

Also, in the disclosure, the expressions “A or B,” “at least one of Aand/or B,” or “one or more of A and/or B,” and the like, may include allpossible combinations of the listed items. For example, “A or B,” “atleast one of A and B,” or “at least one of A or B” may refer to all ofthe following cases: (1) including at least one A, (2) including atleast one B, or (3) including at least one A and at least one B.

In addition, the expressions “first,” “second,” and the like, used inthe disclosure may be used to describe various elements regardless ofany order and/or degree of importance. Also, such expressions are usedto distinguish one element from another element, and are not intended tolimit the elements.

Further, the description in the disclosure that one element (e.g., afirst element) is “(operatively or communicatively) coupled with/to” or“connected to” another element (e.g., a second element) should beinterpreted to include both the case where the one element is directlycoupled to the another element, and the case where the one element iscoupled to the another element through still another element (e.g., athird element). In contrast, the description that one element (e.g., afirst element) is “directly coupled” or “directly connected” to anotherelement (e.g., a second element) can be interpreted to mean that stillanother element (e.g., a third element) does not exist between the oneelement and the another element.

Also, the expression “configured to” used in the disclosure may beinterchangeably used with other expressions such as “suitable for,”“having the capacity to,” “designed to,” “adapted to,” “made to,” and“capable of,” and the like. The term “configured to” may not necessarilymean that a device is “specifically designed to” in terms of hardware.Instead, under some circumstances, the expression “a device configuredto” may mean that the device “is capable of” performing an operationtogether with another device or component. For example, the phrase “asub-processor configured to perform A, B, and C” may mean a dedicatedprocessor (e.g., an embedded processor) for performing the correspondingoperations, or a generic-purpose processor (e.g., a central processingunit (CPU) or an application processor) that can perform thecorresponding operations by executing one or more software programsstored in a memory device.

In the disclosure, the term “user” may refer to a person who uses anelectronic apparatus or an apparatus using an electronic apparatus(e.g., an artificial intelligence electronic apparatus).

Also, the term “subject” used in the disclosure may mean an object whichis the subject of photographing. That is, the term may mean an objectwhich is in a state of being photographed by a camera and not displayedon a display. Also, the term “object” may mean an object included in aphotographed image. That is, the term may mean a state wherein a subjectphotographed by a camera is displayed on a display. Also, the term“virtual object” may mean an object that does not actually exist, but isgenerated as if it exists in a photographed image.

Hereinafter, the disclosure will be described in detail with referenceto the drawings.

FIG. 1 is a diagram for schematically illustrating an electronicapparatus according to an embodiment.

The electronic apparatus 100 may photograph a space around a userthrough a camera. In this case, the electronic apparatus 100 provides animage corresponding to the space around the user to the user.

The electronic apparatus 100 may replace an object 10 included in thephotographed image with a virtual object 20 and generate an augmentedreality image. Here, the object corresponds to an object that actuallyexists in a space around the user, and the virtual object 20 correspondsto a virtual object that does not exist in a space around the user. Forexample, in case in which an air conditioner 10 is included in an imagephotographed by the electronic apparatus 100, the electronic apparatus100 may replace the air conditioner 10 included in the photographedimage with a virtual air purifier 20 that does not exist in a spacearound the user, generate an augmented reality image, and provide thegenerated augmented reality image to the user.

In FIG. 1 , a smartphone is illustrated as the electronic apparatus 100,but the electronic apparatus 100 is not necessarily limited thereto. Inthe disclosure, the electronic apparatus 100 may be implemented as anapparatus equipped with a display function such as a television (TV), atablet personal computer (PC), a portable media player (PMP), a personaldigital assistance (PDA), a laptop, a smart watch, a head mounteddisplay (HMD), and a near eye display (NED). Also, the electronicapparatus 100 may include various forms of displays 120 to provide adisplay function.

The example embodiments of the disclosure may also be implementedthrough an electronic apparatus that is not equipped with a displayfunction. For example, various types of electronic apparatuses providingan image to an external apparatus such as a Blu-ray player, a digitalversatile disc (DVD) player, a streaming content output apparatus, and aset-top box may implement the various embodiments of the disclosure. Asanother example, various forms of home appliances such as a speaker, arefrigerator, a washing machine, an air conditioner, an air purifier,and various kinds of Internet of Things apparatuses may implement thevarious embodiments of the disclosure. Hereinafter, for convenience ofexplanation, explanation will be made based on the assumption that theelectronic apparatus 100 is a user terminal apparatus equipped with acamera 110 and a display 120.

FIG. 2 is a flow chart for illustrating a control method of anelectronic apparatus according to an embodiment.

First, in the control method of an electronic apparatus according to anembodiment of the disclosure, an image is photographed at operationS210. Here, the electronic apparatus 100 may photograph an area opposingthe electronic apparatus 100 via a camera arranged on the rear surfaceof the electronic apparatus 100.

Then, a first guideline and the photographed image are displayedtogether at operation S220. Specifically, the electronic apparatus 100may provide a guide such that the corner of a photographed object isplaced on the first guideline by using the first guideline, and therebyinduce a user to photograph a subject based on the first guideline whenphotographing a subject. Alternatively, the electronic apparatus 100 mayprovide a guide such that an object is photographed while the corner ofthe photographed object is in parallel to the first guideline. As a usermay easily photograph a subject in an accurate angle and an accuratedirection by using the first guideline, and the electronic apparatus 100photographs an object while one surface of the object is in line withthe first guideline, a process of an operation applying an augmentedreality (AR) technology may be simplified. A method of simplifying aprocess of an operation will be described in detail later with referenceto FIG. 8 .

Here, the first guideline is a line displayed on the display 120, and isa user interface (UI) for specifying the location of an object includedin an image. Specifically, the first guideline means a straight line forguiding such that the location of an object included in an image can bethe front surface of the object. Also, on the ½ point of the firstguideline, a line orthogonal to the first guideline is placed, and thecenter of the object may be made to coincide with the line orthogonal tothe first guideline. That is, the first guideline may be a line forguiding such that the camera is toward the front surface of an objectand a vertical center axis of the object can be placed in the center ofthe display 120. However, this is merely an embodiment, and a locationwherein the first guideline is located and whether there is a lineorthogonal to the first guideline may vary.

Then, the electronic device 100 determines whether an object wasrecognized in the photographed image based on the first guideline atoperation S230. In this regard, the electronic apparatus 100 may be setsuch that an object is recognized in the photographed image. Here, anobject corresponds to a subject that exists in reality, and if a subjectis photographed by a camera, an object may be included in thephotographed image. Also, an object is replaced with a virtual object inan augmented reality image, and may be various kinds of home appliancesor external electronic apparatuses. For example, an object may be homeappliances or external electronic apparatuses such as a TV, a monitor, awashing machine, a refrigerator, and an air conditioner. In case theelectronic apparatus 100 photographs an image in real time and displaysthe image on the display 120, the electronic apparatus 100 may recognizean object existing on the display 120 based on the first guideline. Thatis, the electronic apparatus 100 may recognize the size of an object oran area wherein an object exists based on the first guideline.

In addition to the first guideline, a second guideline guiding aphotographing angle of a camera may be generated. The second guidelineindicates a UI for guiding such that a camera photographs an object atan angle parallel to the object. For example, in case a photographingangle of a camera is not perpendicular to the ground surface, i.e., incase a camera does not photograph an object at an angle parallel to theobject, the second guideline may be displayed, and in case a camera isat an angle parallel to an object, the second guideline may not bedisplayed.

Also, according to a photographing angle of a camera, the photographingangle of the camera may be guided by changing the length of the secondguideline. For example, the length of the second guideline may be longerin a case wherein a camera faces an object from the upside than in acase wherein a camera faces an object from the downside.

If a subject is photographed while being aligned with the firstguideline, and the electronic apparatus 100 recognizes an object in thephotographed image based on the first guideline at operation S230-Y, theelectronic apparatus 100 may render a virtual object replacing theobject at operation S240. Here, a virtual object may be an object of thesame kind as the recognized object, but is not necessarily limitedthereto. That is, a virtual object may be various kinds of homeappliances or external electronic apparatuses. Also, a virtual objectmay be a still object, or a moving object.

In addition, a virtual object may be an object selected by a user.Specifically, if an object is recognized, a list including a pluralityof virtual objects that can replace the recognized object may bedisplayed, and if a user input selecting one of the plurality of virtualobjects is received, a virtual object corresponding to the user inputmay be rendered.

The control method of the disclosure may render a virtual object inconsideration of the size of a recognized object and the surroundingblank space of the recognized object. Specifically, the length of ablank space existing in an adjacent area to the recognized object isidentified, and it may be identified whether a virtual object can belocated in the area wherein the object is located based on the lengthinformation of the object and the identified length of the blank space.Then, if it is identified that a virtual object can be located in thearea wherein the object is located, a virtual object may be rendered.

Then, the electronic apparatus 100 may generate an augmented realityimage wherein a rendered virtual object is located in an area wherein anobject is located in the photographed image at operation S250.

Specifically, based on the location information and the angleinformation of a camera photographing an image, the location of avirtual object in an augmented reality image may be identified, and anaugmented reality image wherein a virtual object is arranged in theidentified location may be generated.

To be more specific, based on the location information and the angleinformation of a camera, a movement value and a rotation value of arecognized object may be identified, and based on the movement value andthe rotation value of the object, the size of a virtual object may becalculated and the location of the virtual object may be identified.

It may be identified whether a rendered virtual object can cover anobject area inside an image. Here, the feature that a rendered virtualobject can cover an object area means that a rendered virtual object isoverlapped with an object in an image, and a portion of the object isnot shown.

In a case in which a partial area of an object not covered by a virtualobject exists, the partial object area not covered by the virtual objectmay be replaced with the background area around the partial object areaand an augmented reality image may be generated.

Then, the electronic apparatus 100 may display the generated augmentedreality image at operation S260.

After the generated augmented reality image is displayed, if an eventoccurs, a virtual object indicating a movement during a drivingoperation of a virtual object may be rendered. Here, an event may meanan operation of touching or dragging the rendered virtual object. Forexample, in case a user touches a virtual object (e.g., a refrigerator)in the generated augmented reality image for a predetermined time period(e.g., two seconds), a virtual object indicating a movement during adriving operation of the virtual object (e.g., opening of the door ofthe refrigerator) may be rendered.

A user input moving a virtual object included in an augmented realityimage may be received. In this case, the virtual object may be renderedwhile the location of the virtual object in the photographed image ischanged based on the user input. For example, in case a virtual objectcorresponding to a dryer was rendered beside a washing machine, but auser input moving the virtual object onto the washing machine wasreceived, the virtual object corresponding to the dryer may be renderedto be located on the washing machine.

If the location information of the camera is changed while an augmentedreality image including a virtual object is displayed, the virtualobject may be rendered while being changed based on the changed locationof the camera. For example, in case the location of the camera moves tothe left and right sides of an object while an augmented reality imageincluding a virtual object is displayed, the virtual object displayed onthe image may be rendered while being changed to a virtual objectcorresponding to a left side surface or a right side surface of thevirtual object.

Then, an augmented reality image including the rendered virtual objectmay be displayed.

FIG. 3 is a block diagram for illustrating a configuration of anelectronic apparatus according to an embodiment of the disclosure.

Referring to FIG. 3 , the electronic apparatus 100 includes a camera110, a display 120, and a processor 130. However, the electronicapparatus 100 might not necessarily include a camera 110, a display 120,and a processor 130, and depending on various embodiments, somecomponents among the camera 110, the display 120, and the processor 130may be omitted.

The camera 110 is an apparatus that may photograph a still image or amoving image, and may include at least one image sensor (e.g., a frontsurface sensor or a rear surface sensor), a lens, an image signalprocessor (ISP), and a flash component (e.g., a light emitting diode(LED), a xenon lamp, etc.).

The camera 110 may be located on the front surface or the rear surfaceof the electronic apparatus 100. In the disclosure, explanation will bemade based on the assumption that an image is photographed with a cameraplaced on the rear surface of the electronic apparatus 100, but thedisclosure is not necessarily limited thereto, and an image may bephotographed with a camera placed on the front surface of the electronicapparatus 100.

The camera 110 according to an embodiment may photograph a randomsubject according to control of the processor 130, and transmit thephotographed data to the processor 130. The photographed data may bestored in the memory 150 according to control of the processor 130.Here, the photographed data may referred to as a picture, an image, astill image, and a moving image, but hereinafter, the data will begenerally referred to as an image for the convenience of explanation.Here, a subject may mean an object which is the subject ofphotographing. An image according to the various embodiments of thedisclosure may mean an image received from an external apparatus or anexternal server, or an image stored in the memory 150, etc., other thana live view image photographed through the camera 110.

According to an embodiment, the processor 130 may display an augmentedreality screen through the display 120 based on an image photographedthrough the camera 110.

The display 120 may provide various content screens that can be providedthrough the electronic apparatus 100. Here, a content screen may includevarious content such as an image, a moving image, texts, music, anapplication execution screen, a graphic user interface (GUI) screen,etc.

The display 120 may be implemented as various forms of displays such asa liquid crystal display (LCD), an organic light emitting diodes (OLED)display, a plasma display panel (PDP), a wall, and a micro LED. In thedisplay 120, a driving circuit that may be implemented in forms such asan amorphous silicon (A-Si) thin-film transistor TFT, a low temperaturepoly silicon (LTPS) TFT, and an organic TFT (OTFT), a backlight unit,etc. may also be included together. The display 120 may be implementedas a touch screen combined with a touch sensor, a flexible display, a 3Ddisplay, etc.

Also, the display 120 according to an embodiment may include a displaypanel configured to output an image, and a bezel housing a displaypanel. In particular, a bezel according to an embodiment may include atouch sensor (not shown) for detecting a user interaction.

In particular, the display 120 may display an image photographed in realtime through the camera 110 according to control of the processor 130(e.g., a live view image), an augmented reality (AR) image to which avirtual object is added to the image, or an image acquired by theprocessor 130 based on the image, etc., as will be described below.Here, a virtual object may be referred to as an AR object, a virtualobject image, etc., but hereinafter, it will be generally referred to asa virtual object for the convenience of explanation.

The processor 130 may be electronically connected with the memory 150,and control the overall operations and functions of the electronicapparatus 100. For example, the processor 130 may drive an operatingsystem or an application program, control hardware or softwarecomponents connected to the processor 130, and perform various kinds ofdata processing and operations. Also, the processor 130 may load aninstruction or data received from at least one of other components on avolatile memory and process it, and store various kinds of data in anon-volatile memory.

The processor 130 may be implemented as a dedicated processor (e.g., anembedded processor) for performing the corresponding operation or ageneric-purpose processor (e.g., a central processing unit (CPU) or anapplication processor) that may perform the corresponding operations byexecuting one or more software programs stored in a memory device.

In the disclosure, the processor 130 may be implemented as a digitalsignal processor (DSP) processing digital signals, a microprocessor, anda time controller (TCON). However, the disclosure is not limitedthereto, and the processor 130 may include one or more of a centralprocessing unit (CPU), a micro controller unit (MCU), a micro processingunit (MPU), a controller, an application processor (AP), agraphics-processing unit (GPU) or a communication processor (CP), and anARM processor, or may be defined by the terms. Also, the processor 130may be implemented as a system on chip (SoC) having a processingalgorithm stored therein or large scale integration (LSI), or in theform of a field programmable gate array (FPGA).

The processor 130 may recognize an object in an image photographedthrough a camera. Here, an object may indicate specific home appliancessuch as a TV, a monitor, a washing machine, a refrigerator, and an airconditioner, and such information on a recognizable object may be storedin the memory 150 in advance. An object may mean an object located onthe display 120 in a specific state. Specifically, the processor 130 mayrecognize a photographed object such that one surface of the objectcoincides or is in parallel with the first guideline based on the firstguideline displayed on the display 120.

The processor 130 may generate a user interface (UI) for guiding suchthat a photographed object is located on a line displayed on the display120 or one surface of an object is in parallel with a line displayed onthe display 120, and control the display 120 to display this. Detailedexplanation in this regard will be made with reference to FIGS. 4A, 4B,4C, 5A, and 5B.

Referring to FIGS. 4A, 4B, and 4C, the processor 130 may guide thelocation of an object in an image by using the first guideline 410.

As described above with reference to FIG. 2 , the first guideline 410 isa line displayed on the display, and it may be a UI for guiding thelocation of an object 11 included in an image. That is, the firstguideline 410 indicates a straight line for guiding so that the entirefront surface of the object 11 can be displayed in an image. Forexample, the first guideline may be located on the ⅓ point of thedisplay 120 from the lower end of the display 120. That is, as the firstguideline is located on the ⅓ point from the lower end of the display120, an object included in an image may be guided to be located in thecenter of the display 120. A location wherein the first guideline isdisplayed is not necessarily limited thereto, and the location may vary.

Also, according to an embodiment, for performing recognition of anobject or extracting an object from a background, the electronicapparatus 100 may perform object recognition and extraction by using arelatively more simple mathematical formula and a relatively more simplealgorithm based on the first guideline 410, without using softwaretechnologies with high complexity such as deep learning and computervision. Also, by using the first guideline 410, the electronic apparatus100 may perform object recognition and object extraction without usingdepth data of a space and a camera with a high performance generatingdepth data.

The processor 130 may generate the first guideline 410 for guiding suchthat one surface of an object 11 included in a photographed imagecoincides or is in parallel with a line displayed on the screen of thedisplay 120, and control the display 120 to display the generated firstguideline 410 on the screen of the display 120. For example, theprocessor 130 may provide a guide such that a corner corresponding tothe front surface of the object 11 among a plurality of corners includedin the bottom surface of the photographed object 11 is placed on thefirst guideline 410. Specifically, the processor 130 may control thedisplay 120 to display a straight line 420, which is a straight lineextended from a corner corresponding to the front surface of thephotographed object 11. Then, the processor 130 may provide a guide suchthat the object is photographed while the straight line 420 and thefirst guide line 410 coincide or are in parallel.

As illustrated in FIG. 4A, in case the straight line 420 and the firstguideline 410 are not parallel, the processor 130 may provide a guidesuch that the straight line 420 is parallel with the first guideline410, i.e., the camera 110 is moved so that the camera 110 may face thefront surface of a subject 10 corresponding to an object. Specifically,the processor 130 may control the display 120 to display a message 430-1such as “Face the front surface of the actual product,” provide a guidesuch that the user adjusts the location of the camera 110, and provide aguide such that a corner corresponding to the front surface among thecorners included in the bottom surface of the object 11 is in parallelwith the first guideline 410 in accordance thereto. The electronicapparatus 100 may guide such that the user can photograph the subject 10by an accurate method by using the first guideline 410 and the message430-1. Then, in case the straight line 420 extended from the cornercorresponding to the front surface of the object 11 is in parallel withthe first guideline, the processor 130 may provide feedback informingthe user that the location of the object 11 is located in a locationparallel with the first guideline 410. Here, the feedback may beprovided in various forms such as indication of a message informing thelocation of the object 11, vibration of the electronic apparatus 100,flickering of the screen, the color of the object 11, and change of thethickness of the shade. FIG. 4A illustrates a drawing wherein, in a casein which the straight line 420 extended from the corner corresponding tothe front surface of the object 11 is parallel with the first guideline,feedback is provided to the user by using vibration.

As illustrated in FIG. 4B, the straight line 420 extended from thecorner corresponding to the front surface of the object 11 may beomitted on the display 120. Also, in a case in which the bottom surfaceof the object 11 does not coincide with the first guideline 410, theprocessor 130 may provide a guide such that the bottom surface of theobject 11 coincides with the first guideline 410, i.e., the camera 110is moved so that the camera 110 can face the front surface of a subject10 corresponding to the object. Specifically, the processor 130 maycontrol the display 120 to display a message 430-2 such as “Make thebottom surface of the actual product coincide with the guideline,”provide a guide such that the user adjusts the location of the camera110, and provide a guide such that a corner corresponding to the frontsurface among the corners included in the bottom surface of the object11 coincides with the first guideline 410 in accordance thereto. Theelectronic apparatus 100 may provide a guide such that the user canphotograph the subject 10 by an accurate method by using the firstguideline 410 and the message 430-2. Then, in case the cornercorresponding to the front surface among the corners included in thebottom surface of the object 11 coincides with the first guideline 410,the processor 130 may provide feedback informing the user that thelocation of the object 11 is located on the first guideline 410. Here,the feedback may be provided in various forms such as an indication of amessage informing the location of the object 11, vibration of theelectronic apparatus, flickering of the screen, and change of the color,the shade, and the thickness of the object 11. FIG. 4B illustrates adrawing wherein, in a case in which the corner corresponding to thefront surface among the corners included in the bottom surface of theobject 11 coincides with the first guideline, a feedback is provided tothe user by using vibration.

The processor 130 may guide such that a straight line 440 orthogonal tothe first guideline is placed on the ½ point of the first guideline 410,and the center axis of the object included in the image coincides withthe straight line 440.

Also, in case the center axis of the object 11 coincides with or isadjacent to the straight line 440, the processor 130 may providefeedback informing that the center axis of the object 11 coincides withor is adjacent to the straight line 440 to the user. The feedback inthis case may also be provided in various forms such as indication of amessage informing the location of the object 11, vibration of theelectronic apparatus, flickering of the screen, and change of the color,the shade, and the thickness of the object 11.

As described above, the processor 130 may provide the first guideline,and provide a guide such that the camera 110 is located on the frontsurface of the subject 10 corresponding to the object 11 replaced withthe virtual object 20.

Explanation with reference to FIG. 4A was made based on an embodimentwherein the first guideline 410 and the straight line 420 are parallel,but the explanation may also be applied to a case wherein the firstguideline 410 and the straight line 420 coincide. Likewise, explanationwith reference to FIG. 4B was made based on an embodiment wherein thefirst guideline 410 and the bottom surface of the object 11 coincide,but the explanation may be applied to a case wherein the first guideline410 and the bottom surface of the object 11 are parallel.

As illustrated in FIG. 4C, the processor 130 may guide such that theuser draws a corner corresponding to the front surface among the cornersincluded in the bottom surface of the object 11. Specifically, theprocessor 130 may provide a UI guiding such that the user draws a cornerincluded in the bottom surface of the object 11. For example, theprocessor 130 may control the display 120 to display a message 460 suchas “Draw the bottom surface line of the actual product,” and provide aguide such that the user draws the bottom surface line 450 of the object11. Then, the processor 130 may receive input of the bottom surface line450 of the object 11 from the user, and identify the input bottomsurface line 450. Then, the processor 130 may control the display 120 todisplay a virtual object based on the identified bottom surface line450. Specifically, the processor 130 may recognize the object 11 basedon the bottom surface line 450. That is, the input bottom surface line450 may be a basis for recognizing the object 11 as an alternative tothe first guideline 410. As the bottom surface line 450 may perform therole of the first guideline 410 described above with reference to FIGS.4A and 4B, in this case, the processor 130 may include the firstguideline 410 and the straight line 440, or the first guideline 410 andthe straight line 440 may be omitted. FIG. 4C illustrates a drawingincluding the first guideline 410 and the straight line 440.

Then, the processor 130 may recognize the object 11 based on the bottomsurface line 450, and the processor 130 may calculate informationrelated to the rotation angle and information related to the location ofthe recognized object 11, and render a virtual object replacing theobject 11.

The user may easily follow the guide with input of the bottom surfaceline 450, without having to control the angle of the camera 110 suchthat the corner corresponding to the front surface among the cornersincluded in the bottom surface of the object 11 is in parallel orcoincides with the first guideline 410. For the convenience ofexplanation, FIG. 4C illustrates a drawing wherein the front surface ofthe object 11 is photographed parallel to the first guideline 410, andthe user inputs the bottom surface line 450 into the front surface ofthe photographed object 11. However, it is obvious that, in case thefront surface of the object 11 is not photographed to be in parallel orcoincide with the first guideline 410, the photographed object 11 can berecognized with input of the bottom surface line 450.

The processor 130 may guide the location of the camera, and the angle ofthe camera.

FIGS. 5A and 5B are diagrams for illustrating a second guideline guidingan angle of a camera according to an embodiment.

The processor 130 may acquire the pose information of a camera includingthe rotation information of the camera by using a simultaneouslocalization and mapping (SLAM) algorithm, and identify thephotographing angle of the camera 110 based on the acquired poseinformation of the camera. Detailed explanation in this regard will bemade with reference to FIG. 8 .

Also, the processor 130 may generate a second guideline for guiding thephotographing angle of the camera 110 based on the identifiedphotographing angle of the camera 110, and change the length of thesecond guideline according to the photographing angle of the camera 110.

In addition, the processor 130 may provide a guide such that the angleof the camera 110 located on the front surface of a subject 10 isadjusted by using the generated second guideline. Specifically, theprocessor 130 may change the length of the second guideline according tothe photographing angle of the camera 110, and thereby guide such thatthe camera 110 photographs the subject 10 at an angle parallel to thesubject 10.

FIG. 5A is a diagram illustrating the object 11 and the second guideline510 displayed on the display 120 in case the camera 110 photographed thesubject 10 at an angle facing the ground surface, i.e., at an anglefacing the lower end of the subject 10. In FIGS. 5A and 5B, the line 520may correspond to the first guideline 410 described above with referenceto FIGS. 4A, 4B, and 4C. Here, the line 520 may include the straightline 440 in FIG. 4A, and the straight line 440 may be omitted dependingon cases. FIGS. 5A and 5B illustrate drawings wherein the straight line440 in FIG. 4A is omitted.

As illustrated in FIG. 5A, in case the camera 110 photographed thesubject 10 at an angle facing the ground surface, the processor 130 maycontrol the display 120 to display the second guideline 510 based on theangle of the camera 110.

The processor 130 may change the length of the second guideline 510 suchthat the length of the second guideline 510 becomes longer as thephotographing angle of the camera 110 becomes larger, i.e., the more thecamera 110 faces the ground surface. In contrast, the processor 130 maychange the length of the second guideline 510 such that the length ofthe second guideline 510 becomes shorter as the photographing angle ofthe camera 110 becomes smaller, i.e., the more the camera is located inparallel with the subject 10.

FIG. 5B is a diagram illustrating the object 11 and the second guideline510 displayed on the display 120 in case the camera 110 photographed thesubject 10 at an angle facing the ceiling, i.e., at an angle facing theupper end of the subject 10.

The processor 130 may change the length of the second guideline 510 suchthat the length of the second guideline 510 becomes longer as thephotographing angle of the camera 110 becomes bigger, i.e., the more thecamera 110 faces the ceiling. In contrast, the processor 130 may changethe length of the second guideline 510 such that the length of thesecond guideline 510 becomes shorter as the photographing angle of thecamera 110 becomes smaller, i.e., the more the camera is located inparallel with the subject 10.

The processor 130 may respectively vary the angles that the secondguideline 510 constitutes with the line 520, for distinguishing a casewherein the camera 110 faces the ground surface and a case wherein thecamera 110 faces the ceiling. For example, in case the camera 110 facesthe ground surface, the processor 130 may set the angle that the secondguideline 510 constitutes with the line 520 as a first angle (e.g., 60degrees), and in case the camera 110 faces the ceiling, the processor130 may set the angle that the second guideline 510 constitutes with theline 520 as a second angle (e.g., 30 degrees).

In case the camera 110 is located at an angle parallel to the subject10, the processor 130 may provide feedback informing that the angle ofthe camera 110 is in parallel with the subject 10. The feedback in thiscase may be provided in various forms such as indication of a messageinforming the angle of the camera 110, vibration of the electronicapparatus, flickering of the screen, and change of the color, the shade,and the thickness of the object 11 displayed on the screen.

The processor 130 may use various UIs in addition to the secondguideline and the line 520 for guiding the photographing angle of thecamera. For example, the processor 130 may display the angle of thecamera, or display a message indicating the angle of the camera in onearea of a photographed image. Alternatively, the processor 130 maydisplay a message 530 which is “Align the camera in parallel with theactual product” until the angle of the camera 110 is located at an angleparallel to the subject 10.

Returning to FIG. 3 , in case an object included in a photographed imageis located on the first guideline displayed on the display 120, or animage is photographed with the angle of the camera 110 in parallel withthe subject 10, the processor 130 may recognize an object included inthe photographed image.

Then, the processor 130 may render a virtual object 20 replacing therecognized object. Here, the virtual object 20 may correspond to variouskinds of external electronic apparatuses or home appliances. The virtualobject 20 may be an object corresponding to the recognized object, butis not necessarily limited thereto. That is, in case a recognized objectis an air conditioner, the virtual object 20 may be an air conditionerof a different kind from the recognized object, but it may also be anelectronic apparatus such as a refrigerator and an air purifier whichare not air conditioners.

Based on an object being recognized, the processor 130 may providevarious virtual objects that may replace the recognized object, andrender a virtual object selected according to a user input selecting oneamong the virtual objects.

In this regard, FIGS. 6A and 6B are diagrams for illustrating anelectronic apparatus providing various virtual objects according to anembodiment.

As illustrated in FIG. 6A, based on an object 11 being recognized in aphotographed image, the processor 130 may control the display 120 todisplay a plurality of virtual objects 611, 612, and 613 that mayreplace the recognized object 11. Here, the plurality of virtual objectsmay include the images of the virtual objects that can replace therecognized object, and information related to the virtual objects suchas the serial numbers, the colors, and the release years of the virtualobjects.

The processor 130 may control the display 120 to display various virtualobjects according to a user interaction touching or dragging theplurality of virtual objects 611, 612, 613. For example, while thevirtual object 612 is displayed in the center, and the virtual object611 and the virtual object 613 are displayed on the left and right sidesof the virtual object 612, based on a user input passing the virtualobject 612 to the right being received, the processor 130 may cause thevirtual object 611 to be displayed in the center of the screen. Also, onthe left side of the virtual object 611, a new virtual object (notshown) may be displayed.

The processor 130 may display a list 620 related to the plurality ofvirtual objects in addition to the plurality of virtual objects 611,612, and 613. Also, based on a user input selecting one of a pluralityof list information (e.g., a TV, a washing machine, a refrigerator,etc.) included in the list 620 being received, the processor 130 mayrender a plurality of virtual objects corresponding to the user input.For example, based on a user input selecting a refrigerator among theplurality of list information included in the list 620 being received,the processor 130 may render a plurality of virtual objectscorresponding to the refrigerator.

Based on a user input selecting one of the plurality of virtual objects611, 612, and 613 being received, the processor 130 may render a virtualobject corresponding to the user input. For example, based in the userselecting the virtual object 612 among the plurality of virtual objects611, 612, and 613, as illustrated in FIG. 6B, the processor 130 mayrender the selected virtual object 612, and control the display 120 todisplay a virtual object 20 corresponding to the selected virtual object612 in the location wherein the object 11 is located in the photographedimage. That is, the processor 130 may control the display 120 to displayan augmented reality image replacing the object 11 with the virtualobject 20.

Then, the processor 130 may inform the user that the object 11 wasreplaced with the virtual object 20 in the augmented reality image.Specifically, the processor 130 may control the display 120 to display amessage 630 such as “Replaced with an AR product,” and thereby informthat the object 11 was replaced with the virtual object 20 in theaugmented reality image.

The processor 130 may render the virtual object in consideration of thesurrounding space of the recognized object.

In this regard, FIGS. 7A, 7B, 7C and 7D are diagrams for illustrating anelectronic apparatus rendering a virtual object in consideration of thesurrounding space of a recognized object according to an embodiment ofthe disclosure.

Referring to FIG. 7A, a drawing wherein a recognized object 710 and ablank space 70 existing in the surroundings are displayed on the display120 is illustrated.

The processor 130 may identify a blank space 70 existing in thesurroundings of the recognized object 710. Here, the blank space 70means a space wherein no object or matter exists among the surroundingspaces of the recognized object.

Referring to FIG. 7B, the processor 130 may recognize an object 710 in aphotographed image and recognize a blank space 70 existing in thesurroundings of the object 710, and identify the length of the blankspace 70. Specifically, the processor 130 may identify the length of theblank space 70 identified to exist in an adjacent area of the object 710in the photographed image by using a simultaneous localization andmapping (SLAM) algorithm. Also, the processor 130 may identify thelength information of the recognized object 710 included in thephotographed image by using the SLAM algorithm. Then, the processor 130may identify whether the virtual object 700 can be located in the areawherein the recognized object 710 is located based on the identifiedlength information of the object 710 and length information of the blankspace 70.

The processor 130 may acquire the length information of the actualelectronic apparatus corresponding to the virtual object 700.Specifically, the processor 130 may acquire the length information ofthe actual electronic apparatus corresponding to the virtual object 700from an external apparatus such as a server (not shown). Alternatively,the processor 130 may store, in advance, the length information of theactual electronic apparatus corresponding to the virtual object 700together with the virtual object 700.

As a result of determining the length information of the object 710, thelength information of the blank space 70, and the length information ofthe virtual object 700, if it is identified that the virtual object 700can be located in the area wherein the object 710 is located, then theprocessor 130 may render the first virtual object 720 in the areawherein the object 710 is located as illustrated in FIG. 7C.

Alternatively, as a result of determining the length information of theobject 710, the length information of the blank space 70, and the lengthinformation of the virtual object 700, if it is identified that thevirtual object 700 cannot be located in the area wherein the object 710is located, the processor 130 may inform that an actual productcorresponding to the virtual object 700 cannot be arranged.

Specifically, the processor 130 may compare the length information ofthe actual electronic apparatus corresponding to the second virtualobject 730 with the length information of the object 710 and the lengthinformation of the blank space 70. If the length information of theactual electronic apparatus corresponding to the second virtual object730 is bigger than the numerical value summing up the length informationof the object 710 and the length information of the blank space 70, theprocessor 130 may identify that a product corresponding to the secondvirtual object 730 cannot be arranged in the area wherein the object 710is located.

Referring to FIG. 7D, the processor 130 may render the second virtualobject 730 instead of the object 710 in the area wherein the object 710is located, and control the display 120 to display an area 75 whereinthe actual space and the second virtual object 730 overlap.

Then, the processor 130 may inform that a product corresponding to thesecond virtual object 730 cannot be arranged in the area wherein theobject 710 is located. Specifically, the processor 130 may display amessage 740 such as “As the height of the selected virtual electronicproduct is higher than the height of the actual space, arrangement isimpossible. Please consider this at the time of purchase.”

Also, the processor 130 may identify the location of the renderedvirtual object 700 based on the identified location of the object.

In this regard, FIGS. 8A and 8B are diagrams for illustrating anelectronic apparatus determining the location of an object according toan embodiment of the disclosure. With reference to FIGS. 8A and 8B, amethod of reducing operation complexity will be explained based on theassumption of a situation wherein a user is guided to photograph asubject 10 by using the first guideline, and the subject 10 photographedon a display screen is indicated as an object 11.

Referring to FIG. 8A, the processor 130 may identify the location of anobject 11-1 in an augmented reality image based on the locationinformation and the angle information of the camera 110, and identifythe location of a virtual object based on the identified location of theobject 11-1. Here, the camera 110 may be an RGB camera that can acquireRGB data or a 3D camera that can acquire depth data. The explanationbelow will be made based on the assumption of a case of using a 2Dcamera or an RGB camera that can acquire RGB data.

The processor 130 may acquire the pose information of the camera 110 byusing various SLAM techniques such as a feature-based simultaneouslocalization and mapping (SLAM) technique, a direct SLAM technique, anextended Kalman filter (EKF) SLAM technique, a fast SLAM technique, anda large-scale direct monocular (LSD) SLAM technique. Here, the poseinformation of the camera may include the translation information (Tcx,Tcy, Tcz) and the rotation information (Rcx, Rcy, Rcz) of the camera.Alternatively, the pose information of the camera may be, for example,the X (horizontality), Y (verticality), and Z (depth) corresponding tothe location of the camera and/or a 6 degree of freedom (6 DoF) camerapose including a pitch, a yaw, and a roll corresponding to theorientation of the camera.

The processor 130 may acquire information regarding a plane 810-1 onwhich the subject 10 is placed. Also, the processor 130 may estimate aplane space in a photographed image. Specifically, the processor 130 maycalculate a plane equation by using a 3D point cloud analysis algorithm.For example, the processor 130 may estimate a plane space in aphotographed image by using a random sample consensus (RANSAC) techniquewhich is one of 3D SLAM techniques. Here, the RANSAC technique is amethod of extracting sample data randomly, and obtaining a modelparameter satisfying the extracted sample data.

For example, according to spatial coherency, a plurality of pointsconstituting a plane space may exist while being adjacent to oneanother. The processor 130 may estimate a plane space in an imagesection by using random points adjacent to one another. Also, theprocessor 130 may identify whether the random points adjacent to oneanother exist on the same plane, and acquire information on the planespace based on the identification result.

For example, the processor 130 may identify whether a specific space inan image section is a plane space by using first to third pointsadjacent to one another (or, distanced within a threshold value). If thespecific space is a plane space according to the identification result,the processor 130 may acquire coefficients A, B, C, and D of a planeequation based on the first to third points. Then, the processor 130 mayacquire the plane equation as information on the plane space based onthe following formula 1.Ax+By+Cz+D=0  [Formula 1]

Here, A, B, and C are normal vectors indicating the direction of aplane, and D may be a distance between a plane including the first tothird points and the camera 110. However, this is merely an example, andinformation on a plane space may be in various forms. For example, theprocessor 130 may analyze an image section based on a machine-learnedmodel acquiring feature information of an image, and acquire informationon a plane space according to the analysis result.

The processor 130 may easily calculate the rotation value of the object11-1 from an image photographed based on the first guideline 410.

Specifically, the processor 130 may calculate the rotation values of theobject 11-1 with respect to the x axis and the z axis based on a planeequation calculated based on the above formula 1. For example, theprocessor 130 may acquire a first plane equation (A1x+B1y+C1z+D1=0) fora plane 810-1 on which a subject 10-1 is placed. Then, as the subject10-1 is placed on the plane 810-1, the processor 130 may identify thatthe rotation value Rpx of the object 11-1 for the x axis is A1 which isthe coefficient of x in the first plane equation, and identify that therotation value Rpz of the object 11-1 for the z axis is C1 which is thecoefficient of z in the first plane equation.

Also, the processor 130 may identify the rotation value Rpy of theobject 11-1 for they axis based on the angle information Rcx, Rcy, Rczof the camera 110. Specifically, as the user photographs the subject10-1 on the front surface, the processor 130 may identify that therotation value Rpy of the object 11-1 for they axis is −Rcy.

The electronic apparatus 100 according to the disclosure may easilycalculate the rotation value of the object 11 by using an imagephotographed to be aligned with the first guideline 410.

The processor 130 may identify the coordinate u, v of the crossing point820-1 of the first guideline 410 and the straight line 440 orthogonal tothe first guideline in the image. Here, the identified coordinate u, vof the crossing point 820-1 is a coordinate on the first guideline 410,and may be fixed on a specific location of the display screen.Accordingly, the processor 130 may identify the identified coordinate u,v of the crossing point 820-1.

The processor 130 may acquire information on a straight line passingthrough a virtual location 830-1 and the crossing point 820-1corresponding to the location and angle information of the camera byusing the location and angle information of the camera acquired throughthe identified coordinate u, v of the crossing point 820-1, theintrinsic parameter K of the camera, and the SLAM algorithm. Here, thevirtual location 830-1 may mean a coordinate on an actual spacecorresponding to the location information of the camera. Specifically,the virtual location 830-1 may be a point existing on a straight lineconnecting one point on the subject 10-1 and one point on the object11-1 corresponding to the point, and the virtual location 830-1 may meana location calculated based on the focal length, the focal direction,the photographing angle, the angle information Rcx, Rcy, Rcz, thedistance to the subject, etc. of the camera 110.

Specifically, the processor 130 may acquire an equation regarding astraight line passing through the virtual location 830-1 and thecrossing point 820-1 as information regarding a straight line by usingthe following formula 2.

$\begin{matrix}{{{s{K^{- 1}\begin{bmatrix}u \\v \\1\end{bmatrix}}} - T^{c}} = {R^{c}\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w}\end{bmatrix}}} & \left\lbrack {{Formula}2} \right\rbrack\end{matrix}$

Then, based on the equation regarding the plane 810-1 and the equationregarding the straight line passing through the virtual location 830-1and the crossing point 820-1, the processor 130 may calculate thecoordinate Tpx, Tpy, Tpz of the crossing point 840-1 of the straightline passing through the virtual location 830-1 and the crossing point820-1 and the plane 810-1. Here, the coordinate Tpx, Tpy, Tpz of thecrossing point 840-1 may indicate the center point of the cornercorresponding to the front surface among the corners included in thebottom surface of the recognized object 11-1 in the photographed image.That is, the processor 130 may easily perform the calculation in theabove formula 2 by using the first guideline 410 or the coordinate u, von the first guideline. Then, the processor 130 may alleviate thecomplexity of the operation obtaining an equation regarding the straightline by making the bottom surface of the subject 10 and the firstguideline coincide.

Then, the processor 130 may identify that the subject 10-1 is located onthe coordinate Tpx, Tpy, Tpz of the crossing point 840-1. Then, theprocessor 130 may generate an augmented reality image wherein a virtualobject 20 replacing the object 11-1 is arranged in the identifiedlocation. Specifically, the processor 130 may render the virtual object20 such that the center point of the corner corresponding to the frontsurface among the plurality of corners included in the bottom surface ofthe virtual object 20 is located on the coordinate Tpx, Tpy, Tpz of thecrossing point 840-1.

The processor 130 may calculate the size of the virtual object based onthe virtual location 830-1 corresponding to the location informationTcx, Tcy, Tcz of the camera and the coordinate Tpx, Tpy, Tpz of thecrossing point 840-1. Specifically, the virtual location 830-1corresponding to the location information Tcx, Tcy, Tcz of the cameramay mean the location of the virtual camera, and thus the processor 130may identify the size of the virtual object to be inversely proportionalto the distance between the virtual location 830-1 corresponding to thelocation information Tcx, Tcy, Tcz of the camera and the coordinate Tpx,Tpy, Tpz of the crossing point 840-1. For example, if the distancebetween the virtual location 830-1 and the crossing point 840-1increases as much as a predetermined distance, the size of the virtualobject may be decreased as much as a predetermined size proportion. In acase in which the distance between the virtual location 830-1 and thecrossing point 840-1 is 0, the virtual object may be implemented as theactual size of the product corresponding to the virtual object. Then,the processor 130 may render the virtual object in the identified size.

FIG. 8A illustrates the subject 10-1 placed on a floor, but asillustrated in FIG. 8B, the processor 130 may render the virtual object20 replacing the photographed object 11-2 with respect to the subject10-2 attached to the wall surface based on the same method.

Specifically, the processor 130 may identify the location of the object11-2 based on the location information and the angle information of thecamera 110, and identify the location of the virtual object based on theidentified location of the object 11-2. Also, the processor 130 mayacquire pose information of the camera 110 by using various SLAMtechniques. That is, the processor 130 may use the same technique ormethod for identifying the location of the object 11-2.

The processor 130 may calculate the rotation value of the object 11-2with respect to the X axis and the Y axis based on the plane equationcalculated based on the formula 1. For example, the processor 130 mayacquire a second plane equation (A2x+B2y+C2z+D2=0) for the plane 810-2on which the subject 10-2 is located. Also, as the subject 10-2 islocated on the plane 810-2, the processor 130 may identify that therotation value Rpx of the object 11-2 for the X axis is A2 which is thecoefficient of x in the second plane equation, and identify that therotation value Rpy of the object 11-1 for they axis is B2 which is thecoefficient of y in the second plane equation.

Also, the processor 130 may identify the rotation value Rpz of theobject 11-2 for the z axis based on the angle information Rcx, Rcy, Rczof the camera 110. Specifically, as the user photographs the subject10-2 from the front surface, the processor 130 may identify that therotation value Rpz of the object 11-2 for the z axis is −Rcz.

The crossing point 820-2, the virtual location 830-2, and the crossingpoint 840-2 illustrated in FIG. 8B may respectively correspond to thecrossing point 820-1, the virtual location 830-1, and the crossing point840-1 illustrated in FIG. 8A. The processor 130 may identify thecoordinate u, v of the crossing point 820-2 by the same method ortechnique as described with reference to FIG. 8A, and acquireinformation regarding the straight line passing through the virtuallocation 830-2 and the crossing point 820-2 corresponding to thelocation and angle information of the camera by using the location andangle information of the camera acquired through the intrinsic parameterK of the camera and the SLAM algorithm.

The processor 130 may generate an augmented reality image wherein arendered virtual object is located in an area wherein an object islocated in a photographed image, and control the display 120 to displaythe generated augmented reality image.

As described above, according to an embodiment, the electronic apparatus100 may identify the location of an object in a photographed image byusing only an RGB camera or a 2D camera that can acquire RGB data,without using a camera with a high performance that can acquire depthdata. Also, the electronic apparatus 100 may calculate the location, thesize, and the rotation value of an object based on the simple formula 1and formula 2 without using a complex algorithm in object extraction andobject recognition processes. Accordingly, the real time quality of anaugmented reality image can be improved.

In a case in which a virtual object covers a recognized object in animage, additional image processing might not be necessary. However, incase a virtual object cannot cover a recognized object in an image,additional image processing may be necessary.

FIGS. 9A and 9B are diagrams for illustrating an electronic apparatuswhich performs image processing in case a rendered virtual object cannotcover a recognized object according to an embodiment.

The processor 130 may identify the location and the size of a virtualobject as described above with reference to FIG. 8 , and identifywhether a virtual object rendered in an image can cover an object areain the image. Here, the feature that a rendered virtual object can coveran object area means that a virtual object is totally overlapped with arecognized object, and a portion of the object is not shown in an image.

In case a partial object area not covered by a virtual object exists,the processor 130 may generate an augmented reality image by replacingthe partial object area with the background area around the partialobject area. Here, the background area around the partial object areameans a background area adjacent to the partial area of the object notcovered.

Specifically, the processor 130 may remove the partial object areathrough a vision recognition technology, and replace the removed portionbased on the background area adjacent to the removed partial objectarea.

For example, as illustrated in FIG. 9A, it will be assumed that anobject 11 was recognized in an image photographed by the camera 110.Here, the object 11 is an air conditioner, and a background such as awindow may exist on the rear side of the object 11. Then, it will beassumed that the processor 130 rendered an air purifier as a virtualobject 20 replacing the recognized object 11. In case the air purifierwhich is the virtual object 20 does not cover the air conditioner whichis the object 11 in the photographed image, i.e., in case a partial areaof the air conditioner exists in an augmented reality image, theprocessor 130 may remove the partial area of the air conditioner notcovered in the augmented reality image, and replace the removed areabased on the window, the wall of the glass window, etc. which are thebackground area adjacent to the removed partial object area. FIG. 9Billustrates an augmented reality image wherein the processor 130replaced a removed partial object area 12 based on the window, the wallof the glass window, etc. which are the background area adjacent to theremoved partial object area 12.

FIGS. 10A and 10B are diagrams for illustrating an electronic apparatuswhich changes a virtual object based on the movement of a camera afterthe virtual object was rendered, and renders the virtual objectaccording to an embodiment of the disclosure.

If the location information or the angle information of the camera 110is changed while an augmented reality image is displayed, the processor130 may change a virtual object based on the changed locationinformation or angle information of the camera and render the virtualobject.

For this, the processor 130 may acquire information on the viewdirection (or the view angle) of the camera 110 in the augmented realityimage from the camera 110.

Information on the view direction (or the view angle) of the camera 110according to an embodiment of the disclosure may mean at least one ofthe direction that the camera 110 faces which corresponds to a real-timeimage received from the camera 110, the angle of the camera 110, thelocation of the camera 110, or the posture of the camera 110. Forexample, information on the view direction of the camera 110 may meanthat the camera performed photographing while moving from left to right,or photographed a subject in a specific angle or posture.

According to an embodiment of the disclosure, the electronic apparatus100 may acquire information on the view direction and view angle of thecamera by using at least one of an acceleration sensor or a gyro sensor.For example, the processor 130 may acquire a specific angle (e.g., 15degrees among 360 degrees in all directions) corresponding to the viewdirection that the camera 110 faces (or, the direction that theelectronic apparatus 100 faces) among all directions (e.g., 360 degrees)as information on the view direction based on a sensing result using atleast one of an acceleration sensor or a gyro sensor.

This is merely an example, and the disclosure is not limited thereto.For example, the electronic apparatus 100 can obviously acquireinformation on the view direction of the camera 110 based on a GPSsensor, a tilt/gravity sensor, a location-based services (LBS) method,location information or direction information received from an externalapparatus, a digital compass, etc.

The processor 130 may change a rendered virtual object to a virtualobject corresponding to the acquired view direction information. Thememory 150 may store a plurality of virtual objects related to a virtualobject, and the processor 130 may change the virtual object by using theplurality of virtual objects stored in the memory 150. Here, theplurality of virtual objects related to the virtual object may includeimages wherein an electronic apparatus corresponding to the virtualobject was photographed in various angles and directions. Then, in casethe processor 130 identified that the view direction information of thecamera 110 was changed, the processor 130 may render the virtual objectrendered in an augmented reality image as a virtual object correspondingto the view direction information among the plurality of virtual objectsrelated to the virtual object.

For example, as illustrated in FIG. 10A, it will be assumed that, whilethe front surface of an air conditioner which is a virtual object 20 wasrendered, the camera 110 moved by approximately 30 degrees in acounterclockwise direction based on the subject 10. In this case, theprocessor 130 may acquire the view direction information of the camera110, and change the virtual object to a virtual object corresponding tothe acquired view direction information (e.g., a virtual objectincluding an image which moved by approximately 30 degrees in acounterclockwise direction from the front surface) and render thevirtual object. FIG. 10B illustrates an augmented reality image whereina virtual object was changed based on the changed view directioninformation acquired and was rendered.

FIGS. 11A and 11B are diagrams for illustrating an electronic apparatusrendering a virtual object according to an embodiment of the disclosure.

When an event occurs, the processor 130 may render a virtual objectindicating a movement during a driving operation of a virtual object.Specifically, when an event such as receiving a user input touching avirtual object while a virtual object is rendered in an augmentedreality image occurs, the processor 130 may render a virtual objectindicating a movement during a driving operation of a virtual object.Here, a movement during a driving operation of a virtual object is anobject indicating a driving state of an electronic apparatuscorresponding to a virtual object, and may include an object such as thedoor opening operation of a refrigerator (or a washing machine, adryer), an image displaying operation of a TV, and a menu displayingoperation of a display included in an air conditioner (or arefrigerator).

For example, as illustrated in FIG. 11A, while a refrigerator isillustrated in an augmented reality image as a virtual object 20, if auser input touching the refrigerator displayed on the screen isreceived, the processor 130 may render a virtual object 21 wherein thedoor of the refrigerator is opened, as illustrated in FIG. 11B.

For this, the memory 150 may store a virtual object indicating amovement during an operation of a virtual object, and when an eventoccurs, the processor 130 may render a virtual object indicating amovement during an operation of a rendered virtual object.

In the above, it was described that an event receives a user inputtouching a virtual object, but the disclosure is not necessarily limitedthereto. For example, an event may vary depending on embodiments, suchas receiving a voice input of a user, receiving a user input selecting aUI (e.g., a UI indicating viewing of an operation state) displayed in anaugmented reality image, and receiving a user input shaking theelectronic apparatus 100. Also, it may be possible that a predeterminedtime period passes after a virtual object was rendered and a virtualobject indicating a movement during a driving operation of a virtualobject is rendered, without a user input.

FIGS. 12A and 12B are diagrams for illustrating an electronic apparatuschanging the location of a rendered virtual object.

When the processor 130 receives a user input moving a virtual objectincluded in an augmented reality image, the processor 130 may change thelocation of the virtual object based on the user input and render thevirtual object.

For example, it will be assumed that the camera 110 photographs areaswherein a washing machine 30 and a dryer exist, and the processor 130renders a virtual object 20 replacing the photographed dryer, and asillustrated in FIG. 12A, the processor 130 displays an augmented realityimage including the washing machine 30 and the virtual object 20corresponding to a subject existing in reality on the display 120.

Here, if a user input moving the virtual object 20 is received, theprocessor 130 may change the location of the virtual object 20 based onthe user input and render the virtual object 20. For example, if a userinput selecting a dryer which is the virtual object 20 and dragging itto the upper end of the washing machine 30 is received, the processor130 may change the location of the dryer displayed in the augmentedreality image to the upper end of the washing machine 30.

FIG. 13 is a diagram for illustrating a detailed configuration of anelectronic apparatus according to an embodiment of the disclosure.

As illustrated in FIG. 13 , the electronic apparatus 100 may include acamera 110, a display 120, a processor 130, a communication interface140, a memory 150, an input interface 160, and a speaker 170. Among thecomponents illustrated in FIG. 13 , regarding the components overlappingwith the components illustrated in FIG. 2 , detailed explanation will beomitted.

The communication interface 140 is a component for the electronicapparatus 100 to perform communication with an external electronicapparatus (not shown) such as a server. The electronic apparatus 100 mayreceive various data such as information on a recognized object andinformation on a virtual object from an external electronic apparatus(not shown) through the communication interface 140.

Also, the communication interface 140 may include various communicationmodules such as a wired communication module (not shown), a near fieldwireless communication module (not shown), and a wireless communicationmodule (not shown).

Here, a wired communication module is a module for performingcommunication with an external apparatus (not shown) according to awired communication method such as a wired Ethernet. A near fieldwireless communication module is a module for performing communicationwith an external apparatus (not shown) located in a close distanceaccording to a near field wireless communication method such asBluetooth (BT), Bluetooth Low Energy (BLE), and ZigBee methods. Inaddition, a wireless communication module is a module that is connectedto an external network according to a wireless communication protocolsuch as Wireless Fidelity (Wi-Fi) and Institute of Electrical andElectronics Engineers (IEEE) and performs communication with an externalapparatus (not shown) and a voice recognition server (not shown). Otherthan the above, a wireless communication module may further include amobile communication module that is connected with a mobilecommunication network and performs communication according to variousmobile communication standards such as 3rd Generation (3G), 3rdGeneration Partnership Project (3GPP), Long Term Evolution (LTE), LTEAdvanced (LTE-A), and fifth generation (5G) Networks.

The memory 150 is a component for storing various kinds of programs anddata for the operation of the electronic apparatus 100. The memory 150may be implemented as a non-volatile memory, a volatile memory, aflash-memory, a hard disk drive (HDD), or a solid state drive (SDD),etc. Also, the memory 150 may be accessed by the processor 130, andreading/recording/correction/deletion/update, etc. of data by theprocessor 130 may be performed. In the disclosure, the term memory mayinclude the memory 150, a read-only memory (ROM) (not shown) and arandom access memory (RAM) (not shown) inside the processor 130, or amemory card (not shown) mounted on the electronic apparatus 100 (e.g., amicro SD card, a memory stick).

The memory 150 may store information on various virtual objects of theelectronic apparatus 100. Also, according to an embodiment, theelectronic apparatus 100 may store information on an electronicapparatus which the camera 110 photographed and can be recognized as anobject.

The input interface 160 may receive a user input for controlling theelectronic apparatus 100. In particular, the input interface 160 mayreceive input of voice information of a user for a reproduced content,or receive input of a user input selecting a GUI displayed on a display.In the input interface 160, as illustrated in FIG. 13 , a microphone 161for receiving input of a user voice, a touch panel 162 for receivinginput of a user touch using a user's hand or a stylus pen, etc., abutton 163 for receiving input of a user manipulation, etc. may beincluded. However, the example of the input interface 160 illustrated inFIG. 13 is merely an example, and the input interface 160 may beimplemented as another input apparatus (e.g., a keyboard, a mouse, amotion inputter, etc.).

The speaker 170 is a component outputting various kinds of notificationsounds or voice messages as well as various types of audio data forwhich various processing operations such as decoding or amplification,noise filtering, etc. were performed by an audio processor. Inparticular, the speaker 170 may output a response for voice informationof a user as a voice message in the form of a natural language. Acomponent for outputting audio may be implemented as a speaker, but thisis merely an example, and such a component may be implemented as anoutput terminal that can output audio data.

The electronic apparatus 100 might not necessarily include all of theaforementioned components, and some components may be omitted.

Computer instructions for performing the processing operations of theelectronic apparatus 100 according to the aforementioned variousembodiments of the disclosure may be stored in a non-transitorycomputer-readable medium. Computer instructions stored in such anon-transitory computer-readable medium may implement the processingoperations of the electronic apparatus 100 according to theaforementioned various embodiments of the disclosure performed by theaforementioned specific machine when they are executed by a processor ofthe specific machine.

A non-transitory computer-readable medium refers to a medium that storesdata semi-permanently, and is readable by machines. Specifically, theaforementioned various applications or programs may be provided whilebeing stored in a non-transitory computer-readable medium such as a CD,a DVD, a hard disk, a Blu-ray disk, a USB, a memory card, a ROM and thelike.

Also, while embodiments of the disclosure have been shown and described,the disclosure is not limited to the aforementioned specificembodiments, and it is apparent that various modifications may be madeby those having ordinary skill in the technical field to which thedisclosure belongs, without departing from the gist of the disclosure asclaimed by the appended claims. In addition, it is intended that suchmodifications are not to be interpreted independently from the technicalidea or prospect of the disclosure.

What is claimed is:
 1. An electronic apparatus comprising: a display; amemory configured to store instructions; and a processor configured toexecute the instructions to: control a display to display an imagecaptured via a camera, and a first guideline, identify a position of anarea where an object is located in the image and a size of the areacorresponding to a size of the object based on the first guideline,render a virtual object to replace the object, generate an augmentedreality image including the virtual object that is located in the area,and control the display to display the augmented reality image.
 2. Theelectronic apparatus of claim 1, wherein the processor is furtherconfigured to: based on an event, render the virtual object to indicatea movement of the virtual object.
 3. The electronic apparatus of claim1, wherein the processor is further configured to: identify a length ofa blank space existing in an adjacent area of the object, identifywhether the virtual object is capable of being located in the area wherethe object is located based on length information of the object and thelength of the blank space, and based on the virtual object being capableof being located in the area where the object is located, render thevirtual object.
 4. The electronic apparatus of claim 1, wherein theprocessor is further configured to: identify whether the virtual objectis capable of covering an object area of the object in the image, andgenerate the augmented reality image by replacing a partial object areaof the object that is not covered by the virtual object with abackground area adjacent to the partial object area.
 5. The electronicapparatus of claim 1, wherein the processor is further configured to:control the display to display a plurality of virtual objects that arecapable of replacing the object, and based on a user input selecting oneof the plurality of virtual objects being received, render the virtualobject corresponding to the user input.
 6. The electronic apparatus ofclaim 1, wherein the processor is further configured to: based on a linebeing input in association with a surface of the object in the image,determine a location of the virtual object in the augmented realityimage based on the input line.
 7. The electronic apparatus of claim 1,wherein the processor is further configured to: identify a location ofthe virtual object in the augmented reality image based on locationinformation and angle information of the camera, and generate theaugmented reality image by arranging the virtual object in the location.8. The electronic apparatus of claim 1, wherein the processor is furtherconfigured to: generate a second guideline for guiding a capturing angleof the camera, and change a length of the second guideline according tothe capturing angle of the camera.
 9. The electronic apparatus of claim1, wherein the processor is further configured to: based on receiving auser input that moves the virtual object included in the augmentedreality image, change a location of the virtual object based on the userinput and render the virtual object according to the changed location.10. The electronic apparatus of claim 1, wherein the processor isfurther configured to: based on location information or angleinformation of the camera being changed while the augmented realityimage is displayed, change the virtual object based on the locationinformation or angle information of the camera.
 11. A control method ofan electronic apparatus, the method comprising: capturing an image;displaying a first guideline and the image; identifying a position of anarea where an object is located in the image and a size of the areacorresponding to a size of the object based on the first guideline;rendering a virtual object to replace the object; generating anaugmented reality image including the virtual object that is located inthe area; and displaying the augmented reality image.
 12. The controlmethod of claim 11, further comprising: based on an event, rendering thevirtual object to indicate a movement of the virtual object.
 13. Thecontrol method of claim 11, wherein the rendering a virtual objectcomprises: identifying a length of a blank space existing in an adjacentarea of the object; identifying whether the virtual object is capable ofbeing located in the area where the object is located based on lengthinformation of the object and the length of the blank space; and basedon the virtual object being capable of being located in the area wherethe object is located, rendering the virtual object.
 14. The controlmethod of claim 11, wherein the generating an augmented reality imagecomprises: identifying whether the virtual object is capable of coveringan object area of the object in the image; and generating the augmentedreality image by replacing a partial object area of the object that isnot covered by the virtual object with a background area adjacent to thepartial object area.
 15. The control method of claim 11, furthercomprising: displaying a plurality of virtual objects that are capableof replacing the object; and based on a user input selecting one of theplurality of virtual objects being received, rendering the virtualobject corresponding to the user input.
 16. The control method of claim11, further comprising: based on a line being input in association witha surface of the object in the image, determining a location of thevirtual object in the augmented reality image based on the input line.17. The control method of claim 11, further comprising: identifying alocation of the virtual object in the augmented reality image based onlocation information and angle information of a camera; and generatingthe augmented reality image by arranging the virtual object in thelocation.
 18. The control method of claim 11, further comprising:generating a second guideline for guiding a capturing angle of a camera;and changing a length of the second guideline according to the capturingangle of the camera.
 19. The control method of claim 11, furthercomprising: receiving a user input that moves the virtual objectincluded in the augmented reality image; and changing a location of thevirtual object based on the user input and rendering the virtual objectaccording to the changed location.
 20. A non-transitorycomputer-readable medium configured to store instructions that, whenexecuted by a processor of an apparatus, cause the processor to: controla display of the apparatus to display an image captured via a camera ofthe apparatus; control the display to display a first guideline in theimage; identify a position of an area where an object is located in theimage and a size of the area corresponding to a size of the object basedon the first guideline; render a virtual object to replace the object;generate an augmented reality image including the virtual object that islocated in the area; and control the display to display the augmentedreality image.